In FY18-19, the iOS team will develop contribution features of the app (project page).

This is an initial exploration with two goals:

  • Write relevant queries that can be productionized later (made to execute daily with data flowing into Druid/Superset).
  • Learn what the program metrics look like right now so we can set our expectations accordingly.

Metrics

Primary metrics

  • Total iOS app edit counts: Number of non-bot edits which are tagged as made through the iOS app interface.
  • iOS edits per editor: For any given time period (monthly/quarterly), of the non-bot registered users that completed at least 1 edit via the iOS app, how many edits did they complete on average (arithmetic mean/median). Edits on other platform won’t affect this metric.

Health monitoring metrics

  • Revert rate of iOS edits: Proportion of “ios app edit”-tagged contributions which have been reverted
  • iOS active contributor retention rate: Number of active iOS editors who have made additional edits through the iOS app in the 30 days following their first iOS edit in the given time span (does not include editors who have only made additional edits through other platforms following their first iOS edit)
  • iOS new contributor retention rate: Number of new iOS editors who have made additional edits through the iOS app in the 30 days following their first ever iOS edit (does not include editors who have only made additional edits through other platforms following their first iOS edit)

In this report, we will compute the baseline for all languages of Wikipedia, with a focus on our target wikis in the annual plan: French, English, Korean, Hindi and Czech.

Note: Platform-specific edit data was not available until June 29 via T194424. Prior to the deployment of the patch, all edits made in iOS and Android apps were lumped together under a single change tag “mobile app edit”. Since multi-platform usage and platform switching are not common occurrences, any editor who has made a “mobile app edit”-tagged contribution previously is ineligble to be considered a new iOS editor.

Query:

SELECT
ios_app_edit_counts.local_user_id AS local_user_id,
IFNULL(previous_app_editors.new_ios_editor, 'TRUE') AS new_ios_editor,
edit_date,
edits,
content_edits
FROM (
SELECT
local_user_id,
edit_date,
COUNT(*) AS edits,
SUM(namespace = 0) AS content_edits
FROM (
-- Edits made with iOS app on visible pages:
SELECT
rev_user AS local_user_id,
DATE(LEFT(rev_timestamp, 8)) AS edit_date,
page.page_namespace AS namespace
FROM change_tag
INNER JOIN revision ON (
revision.rev_id = change_tag.ct_rev_id
AND revision.rev_timestamp >= '{start_date}'
AND revision.rev_timestamp < '{end_date}'
AND change_tag.ct_tag = 'ios app edit'
)
LEFT JOIN page ON revision.rev_page = page.page_id
UNION ALL
-- Edits made with iOS app on deleted pages:
SELECT
ar_user AS local_user_id,
DATE(LEFT(ar_timestamp, 8)) AS edit_date,
ar_namespace AS namespace
FROM change_tag
INNER JOIN archive ON (
archive.ar_id = change_tag.ct_rev_id
AND archive.ar_timestamp >= '{start_date}'
AND archive.ar_timestamp < '{end_date}'
AND change_tag.ct_tag = 'ios app edit'
)
) AS ios_app_edits
GROUP BY local_user_id, edit_date
) AS ios_app_edit_counts
LEFT JOIN (
-- Editors who have used a mobile app (Android/iOS)
-- to edit Wikipedia before the start_date:
SELECT DISTINCT local_user_id,
'FALSE' as new_ios_editor
FROM (
-- Editors who have previously used a mobile app to edit visible pages:
SELECT rev_user AS local_user_id
FROM change_tag
INNER JOIN revision ON (
revision.rev_id = change_tag.ct_rev_id
AND revision.rev_timestamp < '{start_date}'
AND change_tag.ct_tag = 'mobile app edit'
)
UNION ALL
-- Editors who have previously used a mobile app to edit deleted pages:
SELECT ar_user AS local_user_id
FROM change_tag
INNER JOIN archive ON (
archive.ar_id = change_tag.ct_rev_id
AND archive.ar_timestamp < '{start_date}'
AND change_tag.ct_tag = 'mobile app edit'
)
) AS combined_revisions
) AS previous_app_editors
ON previous_app_editors.local_user_id = ios_app_edit_counts.local_user_id

Total iOS app edit counts

Number of non-bot edits which are tagged as made through the iOS app interface. Bot filter are not included in the query since it’s very unlikely that a bot would use the app.

All Edits


Content Edits

Edit counts in the 0 namespace.


Summary

  • Metrics for all wikis and target groups: total edits on ios, total edits for target group (Note that they’re affected by enwiki largely)
  • content edit is large proportion of all edits
  • list out baseline for each target wiki, and the sum of them
  • Hindi: too little edits, variation is sensitive

iOS edits by editor

For any given time period (monthly/quarterly), of the non-bot registered users that completed at least 1 edit via the iOS app, how many edits did they complete on average (arithmetic mean/median). Edits on other platform won’t affect this metric. Bot filter are not included since it’s very unlikely that a bot would use the app.

Here we started by counting the number of edits per editor in July and August 2018 without doing any aggregation, since we want to see the distribution before determine whether we should use mean or median, or other aggregation metrics.

All Edits



Content Edits



Summary

  • For wikis like Hindi with too small number of editors, variation is sensitive

Metrics Suggestion

From the tables and plots above, the distribution of edits per editor is very right skewed with lots of 1-edit editors, thus we can see the average or median edits per editor are not very descriptive. Therefore, I suggest we:

  • Track the proportion of 1-edit editors among all registered editors who edit at least once via iOS app for any given time period (monthly/quarterly). The smaller this number is, the better.
  • Since we consider editors with at least 5 edits per month as active editors (https://meta.wikimedia.org/wiki/Research:Active_editor, this threshold normally applies to desktop edits, so we should probably find another threshold for mobile editing), we want to track the proportion of 5-edit editors among all registered editors who edit at least once via iOS app monthly. The larger this number is, the better.
  • Since the iOS team’s goal is to improve the experience for existing editors, for “iOS heavy wikis” (we still need to define it clearly), we may want to track the number of editors who edit at least once via iOS app for any given time period (monthly/quarterly).

Revert rate of iOS edits

Non-deleted edits only.

Retention

New iOS editor retention

A new iOS editors is any account that has made at least one edit with the iOS app since – but not any edits with either app prior to – July 1st 2018 to August 31st 2018. This includes users who have previously edited on desktop and/or mobile web.

caveats: a lot of user haven’t reach 60 days window

Active iOS editor retention

An active iOS editors is any account that has made at least one edit with the iOS app in July and August 2018.